A method for generating a 3d physical model of a patient specific anatomic feature from 2d medical images

ABSTRACT

There is provided a method for generating a 3D physical model of a patient specific anatomic feature from 2D medical images. The 2D medical images are uploaded by an end-user via a Web Application and sent to a server. The server processes the 2D medical images and automatically generates a 3D printable model of a patient specific anatomic feature from the 2D medical images using a segmentation technique. The 3D printable model is 3D printed as a 3D physical model such that it represents a 1:1 scale of the patient specific anatomic feature. The method includes the step of automatically identifying the patient specific anatomic feature.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The field of the invention relates to a method for 3D printing a patientspecific anatomic feature based on 2D medical images. More particularly,but not exclusively, it relates to methods and systems for managing theprocess of printing 3D physical models of specific anatomic features,methods and systems for automatically segmenting 2D medical images, andmethods and systems for automatically identifying patient specificanatomic feature from 2D medical images.

A portion of the disclosure of this patent document contains material,which is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

2. Description of the Prior Art

Orthopaedic surgeons currently use traditional 2D radiological imaging(such as CT or MRI) to plan patient specific surgeries, which usuallyinvolve a clinician assessing a large number of 2D radiological imagesand planning operations based around their interpretations of thepatient's injuries.

In 2013, L129.5 million was paid out in litigation costs as a result ofcomplications within orthopaedic surgeries in the NHS. The orthopaedicexpenditure in the NHS was approximately £10 Billion in 2016. Thisrepresents the third largest expenditure within the NHS, being justoutweighed by cardiac and mental health costs. With patients typicallyhaving longer durations of life and with the increasing issue of obesitywithin the UK there will be an ever-expanding amount of patientsrequiring orthopaedic treatment and backed up by the rise of consultantswithin orthopaedics rising by ˜250 in the past 3 years.

3D printing is receiving a great deal of attention, and may be used tocreate custom medical prototypes, such as patient specific implantdevices. 3D printing of anatomical models from medical data wastraditionally based on manual techniques associated with collecting dataand prescriptive information from surgeons. The current availableplatforms available for processing medical data and creating 3D printedmodels are often very technical and require experienced users withextensive knowledge on the segmentation techniques used. In addition,segmentation techniques are also time consuming and are not automatic.

Hence current approaches do not cater to the entire medical market, asthe majority of surgeons are not willing to allocate the software andtraining resources required to process the medical images for eachapplicable case.

There is a need for an easy to use and intuitive system that wouldleverage the advances in 3D printing and 3D visualization. It wouldimprove the effectiveness of the surgeons' preoperative planning byproducing a 3D printed model of a particular patient's injured anatomyin the required timeframe whilst maintaining the highest quality. Such asystem would, in turn, further reduce costs to the healthcare providerby diminishing the chances of patient complications.

SUMMARY OF THE INVENTION

One aspect of the invention is a method for generating a 3D physicalmodel of a patient specific anatomic feature from 2D medical images, inwhich: the 2D medical images are uploaded by an end-user via a WebApplication and sent to a server; the server processes the 2D medicalimages and automatically generates a 3D printable model of a patientspecific anatomic feature from the 2D medical images using asegmentation technique; and the 3D printable model is 3D printed as a 3Dphysical model such that it represents a 1:1 scale of the patientspecific anatomic feature.

The 2D medical images may be uploaded alongside metadata, wherein themetadata is either uploaded or entered by the end-user.

The patient specific anatomic feature may automatically be identifiedfrom analysing the 2D medical images.

The analysis of the 2D medical images may use machine learning appliedto an anatomical knowledge dataset.

The patient specific anatomic feature may automatically be identifiedfrom an analysis of the metadata.

The analysis of the metadata may be done using Natural LanguageProcessing (NLP).

The metadata may include patient's prescription information, medicalprofessional information or any other patient-related additionalinformation.

The metadata may be added by the end-user in real time via the WebApplication.

The 3D printable model may automatically be displayed to the end-uservia the Web Application, and the end-user is able review, annotateand/or modify the 3D printable model in real time.

The 2D medical images may be images from the patient taken from a CT,MRI, PET and/or SPCET scanner.

The 2D medical images and any additional metadata may be anonymisedprior to being sent to the server such that no identifiable healthcareor personal information is being transferred to the server.

A cryptographic hash of the patient information may be created to enableidentification of a specific patient without sharing sensitive patientinformation.

A smart contract object, may be required to order or initiate thegeneration of the 3D physical model, contains information about therequirements of the 3D physical model to be printed including one ormore of: stage quality gates, insurance status, payment status/limits,clinician, patient consent, annotations, data sharing agreements and/ordata processing agreements.

The smart contract object may be incorporated into a Blockchain.

The smart contract object may be pre-agreed between the patient and theend-user.

The printing of the 3D physical model may only be executed once thesmart contract object has been validated.

Digital signatures may be used to verify identity and approve decisions.

Dynamic pricing may be generated after the validation of the smartcontract object, and an instant quotation is displayed to the end-user.

A digital currency may be linked to the printing of the 3D physicalmodel.

The material used for printing may be automatically selected dependingon its resemblance to the specific anatomic feature.

The material used for printing may be automatically selected based onthe requirement of the 3D physical model, such as the time required toachieve the printing of the 3D physical model.

The texture used for printing may be automatically selected depending onthe specific anatomic feature.

The 3D physical model may be optimized based on the following patientrelated parameters: scan type, bone type, tissue type, age, gender,weight.

Patient related parameters may be extracted from data uploaded orentered via the Web Application.

Multiple 3D printable models may be generated and multiple 3D physicalmodels are printed and combined when a single printer cannot print thepatient anatomic feature.

One or more connective pieces may be printed in order for the multiple3D physical models to be combined together, and the material used forprinting the connective pieces is automatically predicted.

The end user may select the specific anatomic feature they wish to 3Dprint.

The segmentation technique may be one or a combination of the followingtechniques: threshold-based, decision tree, chained decision forest orneural network method.

Multiple segmentation techniques may be used, and the results of eachsegmentation technique are combined together to derive a finalsegmentation result.

A threshold-based segmentation method may be used and the thresholdvalue is generated from the 2D medical images histogram analysis.

A threshold-based segmentation method may be used and the thresholdvalue is a function of the type of 2D medical images (CT, MRI, PET orSPCET).

The segmentation technique may use a logistic or probabilistic functionto calculate the likelihood of a pixel of being the tissue correspondingto the patient specific anatomic feature.

A threshold-based segmentation method may be used in combination with apre-processing filter such as a Gaussian filter.

A threshold-based segmentation method may be used and the thresholdvalue is a function of the 2D medical images scanning parameters such asX-Ray energy and/or flux.

A threshold-based segmentation method may be used and the thresholdvalue is a function of the bone type (hard or soft).

A threshold-based segmentation method may be used and the thresholdvalue is optimised based on one or more of the following parameters:scan type, bone type, tissue type, age, gender and weight of thepatient.

A threshold-based segmentation method may be used and the thresholdvalue is generated from the 2D medical images histogram analysis.

A threshold-based segmentation method may be used and the thresholdvalue is generated from detecting the peaks of the 2D medical imageshistogram corresponding to tissues similar to the tissue of the patientspecific anatomic feature.

The segmentation technique may further comprise the following steps: thedetected peak inflection point is derived by calculating the zero of thehistogram second derivative in proximity of each peak; the offsetbetween a peak and the inflection point is derived; the estimatedthreshold corresponds to the position of the peak with an offsetcorresponding to three time the inflection offset.

A threshold-based segmentation method may be used and the thresholdvalue is automatically generated and not selected by the end-user.

A threshold-based segmentation may be used and multiple thresholds areapplied to the 2D medical images such that multiple 3D printable modelsare automatically generated.

The segmentation technique may use a decision tree, in which thefollowing properties of the 2D medical images pixels are selected inorder to create the decision tree: number of pixels resembling thetissue of interest located near the pixel in question, number of pixelsresembling the tissue of interest located near the pixel in question,how strong is the overall gradient of the image at the given pixel ifthe consistency of the gradient direction within a small neighbourhoodof the pixel, and in which the tissue of interest is the tissuecorresponding to the patient specific anatomic feature.

The segmentation technique may use a decision tree, and in which thedecision tree is trained using existing pre-labelled medical images.

The decision tree may be applied to a subset of pixel within theoriginal 2D medical images and the labels obtained from this subset arethen up scaled using standard interpolation methods in order to recoverthe segmentation of the full image.

The subset of pixel may be generated by subsampling the original 2Dmedical images, and in which the subsampling stride is selecteddepending on the pixel size. The segmentation technique may use achained decision forest.

A hierarchy of decision forests may be used, in which the results of adecision tree and the results from another segmentation technique arefed to a new decision tree alongside the original pixel values.

Each forest-node may be treated as a simple classifier that produces ascore as to how likely a pixel is to belong to the tissue correspondingto the specific patient anatomic feature.

The segmentation technique may use a Neural Network method, in which theNeural Network is trained from a database of existing medical images.

The neural network may use a Fully Convolutional Neural Network (FCNN).

The neural network may be a UNET Neural Network.

The Neural Network may include only convolutional, downsampling andupsampling layers.

The Neural Network may not include any fully connected layer andcombines the ideas of uNET and FCNN in order to obtain an optimisedsegmentation in terms of anatomical fidelity regarding the edge of theanatomic feature.

Upsampling layers may be added and in which the outputs of previouslayers are used to identify regions of the 2D medical images in order tolead to a specific classification.

The training of the Neural Network may be performed by using a databaseof existing medical images that have been labelled and a medical imagingontology.

The segmentation of the 2D medical images may be performed to classifyeach pixel within the 2D medical images.

The segmentation step may be combined with an anatomic featureidentification algorithm.

The anatomic feature identification algorithm may use a graph databaseof medical images anatomic features.

The method further may include establishing links between the differentclassified pixels from the exploration of the graph database, andidentifying the patient specific anatomic feature from the establishedlinks.

The graph database may comprise nodes representing anatomic featuressuch as tissue type and/or organ type, and edges associated with therelationships between the nodes, such as: has part, proximity,attachment, ligament, functional.

A node may include: a reference to a medical image with thecorresponding anatomic feature, a reference to the results of thesegmentation of a medical image with the corresponding anatomic feature,information relating to the anatomic feature such as volume, surfacearea, Hounsfield Unit standard deviation or average.

The graph database may be updated after the generation of a 3D printablemodel.

A score or probability that the anatomic feature has been correctlyidentified may be provided.

The method may further include a feature extraction algorithm that takesadvantage of both the segmentation as well as the as the 2D medicalimages data in order to obtain interesting properties of the tissue ororgan corresponding to the patient specific anatomic feature.

The feature extraction algorithm may be used to extract one or more ofthe following: the anatomic feature volume, the anatomic feature surfacearea, the anatomic feature Hounsfield unit, the anatomic featurestandard deviation across the all available scans, histogram of theHounsfield Units corresponding to the anatomic feature across ananatomical knowledge dataset or the smallest bounding box containing theanatomic feature.

The feature extraction algorithm may be used to extract one or more ofthe following: the presence of specific keypoint landmarks, a number ofpredefined shapes and volumes within the anatomic feature beingconsidered or specific features that are unique to the specificanatomical component are detected.

The extracted interesting features may be added to the graph database aspart of the node properties.

The extracted interesting features maybe used in order to derive aclassification of the anatomical components located within the scan,using the following steps: derive accurate segmentation using one ormore automated segmentation techniques; apply the feature extractionalgorithm(s) to the segmentation in order to derive the values of suchfeatures; compare to the existing dataset of interesting features andattempt to find a number of matches; the matches are constrained andfiltered depending on the proximity map derived from the graph database;the standard models are used to further refine the filtering andcross-checking by fitting a linear transform between the semi-classifiedsegmented objects and what the standard model looks like; due to theinherit inaccuracies of the segmentation step, each refinement of thematches produce a score or probability of having matched the anatomicalfeatures correctly; the set of scores obtained can be used in a decisiontree (or forest) in order to derive the final classification for aspecific tissue or organ;

A any deviation from standard dataset may be detected.

Touching organs or tissues may be detected within the 2D medical imagesand an edge finding algorithm is used to separate the different tissuesor organs.

The anatomic feature classification may be re-estimated once thedifferent tissues or organs are separated.

Deformities and/or pathology of the anatomic feature may be detected bymeasuring the deviation from the normal or healthy appearance of theanatomic feature.

The method may be used for generating a 3D No-compatible physical modelof a patient specific anatomic feature or a portion of a patientspecific anatomic feature, in which the automatically segmented data isassessed against statistical model of pre-segmented anatomy ‘Best fitmodel’, and a 3D printable model is created based on a statistical modelfor patients' anatomy to insure an optimal reconstruction of tissue, andin which missing fragments are predetermined with a best fit model andtissue scaffold models created from this.

A 3D surface mesh model of the patient specific anatomic feature may begenerated from the segmented 2D medical images.

The 3D surface mesh may be extracted from the scalar volume data.

The 3D surface mesh model may be processed by a mesh cleaning algorithm.

The 3D surface mesh model may be compressed, smoothed and reduced beforebeing sent back to the end-user via the Web-Application.

The 3D surface mesh model may be 3D printable and has the followingproperties: all disjointed surfaces are closed manifolds; appropriatesupports are used to keep the disjointed surfaces/volumes in place,appropriate supports are used in order to facilitate 3D printing; allsurface volumes are not be hollow; if a hollow volume is specificallyrequested by an end-user: appropriate drainage holes are added manuallyby an operation team.

A 3D model of the surface of the patient specific anatomic feature maybe extracted from the 3D mesh model, and in which a marching cubealgorithm is used in order to force some vertices to be placed on voxelsthat do not intersect the iso-surface directly.

The mesh may be as close to a printable model as possible.

The method further may include a surface conditioning step.

The surface conditioning step may include the following step:poly-reduction algorithms are applied to the 3D mesh, errors such asduplicated points, overlapping surfaces, missing surface are correctedto ensure the mesh is a manifold; a mesh filter is applied; holes aredetected and covered; appropriate textures are selected.

The method may further include the following steps: watertight surfacesare filled; dowels are added to support the printing of a specificanatomic feature; print supports are added by determining all localminima of the surface; the print supports are removed during printingpost processing.

The generation of the 3D printable model may be performed by parallelprocessing.

One or more 3D printable models may be sent to the end-user via theWeb-application.

The end-user may select a 3D printable model he wishes to print.

The method may be configured to detect an anomaly within the 2D medicalimages, such as: incorrect classification of medical images, incorrectpatient data, presence of foreign objects in medical images or lowquality imaging data.

An end-user may be alerted when an anomaly is detected.

The method may be able to handle 2D medical images which includeunwanted artefacts or background noise, such as foreign objects or abed.

A preview of the 3D printable model may be displayed to the end-user forapproval before printing the 3D physical model.

Information on the expected timeframe to generate a 3D physical modelmay be calculated and displayed to the end-user in real time.

The information on the expected timeframe may take into considerationthe segmentation, surface conditioning and printing phases.

The printing of a 3D physical model may be scheduled based on inboundmodels and surgical requirement.

The method may select a printer based on the printer parameter and the3D physical model parameter including one or more of: build volume,materials available, minimum feature size, hollow structures within themodel.

The 2D medical images and any additional metadata may be hard linked tothe 3D physical model via a QR code, NFC chip or RFID tag.

The profile of an end-user may be saved alongside the end-userpreference.

One or more end-users may be able to access the Web application, and inwhich each end-user have their own user preferences and user permissionslevels.

An audit trail of the 3D printing process may be created andcontinuously updated and tracked.

Another aspect is a 3D printable model or file of a patient specificanatomic feature that is generated from any of the above methods.

Another aspect is a 3D physical model representing a 1:1 scale of apatient specific anatomic feature that is generated from any of theabove methods.

Another aspect is a computer implemented system for generating a 3Dprinted model of a patient specific anatomic feature from 2D medicalimages, the system comprising: an interface module configured to receive2D medical images and to send the 2D medical images to a server, aserver configured to process the 2D medical images and automaticallygenerate a 3D printable model of a patient specific anatomic featurefrom the 2D medical images using a segmentation technique; and a 3Dprinter configured to receive the 3D printable model and to 3D print a3D physical model such that it represents a 1:1 scale of the patientspecific anatomic feature.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s),with reference to the following Figures, which each show features of theinvention:

FIG. 1 is a diagram illustrating the Axial3D system.

FIG. 2 is a high-level diagram illustrating three basic segments forcreating a 3D printed medical model.

FIG. 3 shows examples of 3D printed model reproducing a patient'sinjuries, such as calcaneus, pelvis and skull fractures.

FIG. 4 shows a workflow diagram illustrating how surgeons may useAxial3D for their preoperative planning processes.

FIG. 5 is a screenshot of the Web application allowing an end-user toupload patient data and order 3D models quickly and easily.

FIG. 6 is a screenshot of the Web application allowing an end-user toupload patient data and order 3D models quickly and easily.

FIG. 7 is a diagram illustrating the workflow for uploading dataanonymously.

FIG. 8 is a diagram illustrating the workflow from receiving the data todisplaying the data and obtaining 3D models ready to print.

FIG. 9 is a diagram illustrating the main aspects of the workflow.

FIG. 10 is a diagram illustrating the automated segmentation workflow.

FIG. 11 is an example of medical image segmentation using a thresholdestimate.

FIG. 12 shows images displaying the bone threshold estimated using ourmethod for the Hounsfield unit histogram from 24 CT scans.

FIG. 13 is an histogram of the threshold across 175 CT scans.

FIG. 14 is an example of medical image segmentation using random forest.

FIG. 15 is a diagram illustrating the chained decision forest workflow.

FIG. 16 is a graph database containing primary anatomical features.

FIG. 17 are images illustrating image reconstruction using comparison.

DETAILED DESCRIPTION

This Detailed Description section describes one implementation of theinvention, called the Axial3D system.

The Axial3D system provides three-dimensional (3D) printed models foruse in medicine, manufactured using patient data medical images tocreate custom products for use in a wide variety of medicalapplications. More specifically, the Axial3D system provides softwareand services that facilitates the production of bespoke 3D printedanatomical models for use by medical professionals using medical imagesas an input, hence bridging the gap between 3D printing technology andmedicine.

FIG. 1 illustrates how the Axial3D system uses unique visualizationtechniques and accurate anatomical knowledge dataset to create a 3Drepresentation (1) of a patient's scan. Medical images are processed anda 3D model is printed to give a 1:1 scale representation of a patient'sinjuries (2). Hence, the 3D printed model is a reliable modelreproducing the patient's exact anatomy and is designed to improvediagnosis and to help plan a personalised treatment for each individualpatient. The medical images are often generated by standard medicalimaging techniques such as but not limited to CT (ComputerisedTomography) and MRI (Magnetic Resonance Imaging).

FIG. 2 is a diagram illustrating the three high-level segments of theAxial3D system enabling the creation of a 3D printed medical model:

-   -   Sending medical images and prescribing requirements or        additional metadata (21);    -   Segmenting images to create 3D printable file of a patient        specific anatomic feature (22);    -   Managing the print process and sending 3D printed physical        model(s) (23);

The models produced can be used for a wide range of healthcareapplications, such as but not limited to preoperative planning andeducation. Medical professional, such as surgeons, are able to gainaccess to 3D printed models as part of their daily routine in assessingtreatment pathways for their patients. 3D printable files can be createddirectly within the hospital through their pre-existing web applicationwithout the need to learn how to use cumbersome software that will takehours to segment images.

The surgeons are therefore provided with a much more comprehensive ideaof what injury they will be treating, by allowing them to conceptualisethe patient's injury in greater detail therefore increasing theeffectiveness of their surgeries. This results in improved patient carecombined with a reduction in resource burden on the hospital.

FIG. 3 shows examples of 3D printed model reproducing a 1:1 scalerepresentation of a patient's injury, such as calcaneus, pelvis andskull fractures.

The Axial3D system can easily be integrated with the clinical workflows,thus giving medical professionals an easy access to the process of 3Dprinting that is currently detached from the medical world. Establishingthis connection with 3D printing and healthcare is done by automatingand refining the processes and creating intuitive, easy to use tools toseamlessly bridge the gap between 3D printing and healthcare.

FIG. 4 is a diagram representing an example on how surgeons may useAxial3D for their preoperative planning processes in order to speed upsurgery times. In this case, surgeons are able to pre-bend and prepareall necessary equipment required to treat a patient well in advance ofentering the operating theater, a process that is currently completedwhen the patient is in surgery. These new methods can typically savearound 30 minutes of procedure time, which costs the NHS

60 per minute in direct surgical costs (

1800 per procedure). As the patient spends less time in surgery, theyalso experience less intraoperative bleeding and a reduce risk ofinfection. This will then go onto reduce the amount of post-operativecare they will require, which in ICU terms can typically cost

2000-

4700 per day.

The Axial3D system therefore enables healthcare providers to improvepatient care and to reduce overall costs. Patient's understanding ofoperations is also improved and patients are then able to adjust theirperception of the risk of operations.

The Axial3D system fills the gap in the market for providing software tostreamline the integration of 3D printing into hospitals by providingtools and infrastructure, enabling 3D printing to seamlessly integratewith healthcare systems and clinical care pathways. This allows thecustomer to manage the printing process from start to finish, within theaxial3D platform, removing bottlenecks in market adoption.

FIGS. 5 and 6 show screenshots of the Web Application allowing anend-user to upload patient data and order 3D models quickly and easily.The Axial3D software automates all of the steps in the process ofcreating 3D printed pre-operative planning models for use in medicine.The purchase of a model is vastly simplified using the Axial3D systemworkflow management. For the clinician this lowers the barrier topurchasing. The hospitals are also provided with reporting and usermanagement services allowing them to manage their consumption of modelsas a whole. Having a web-first platform enables Axial3D to market itsservices to the global market by mitigating ‘the middle man’ incollecting the patients' data and prescriptive information.

Unique features include, but are not limited to:

-   -   Data science and machine learning techniques are applied to        deliver improved 3D models.    -   Anatomic features are recognised using an anatomical dataset        that improves using manual and/or machine learning    -   The service is directly integrated into the medical field.    -   The need to liaise on a regular basis with the surgeon is        largely eliminated.    -   The workflow guides the user through the order process, from the        input of patient details through to anatomy selection and        delivery.    -   A non-technical person is able to order a 3D model.    -   Surgeons or non-technical person can easily and intuitively        upload a patient's data.    -   The drag and drop interface allows an easy upload of large DICOM        (Digital Imaging and Communications in Medicine) image series.    -   DICOM images that come from a variety of sources (such as but        not limited to CT/MRI/PET) may be processed together.    -   DICOM images which contains unwanted artifacts or background        noise, such as foreign objects or a bed, may still be converted        and processed.    -   Development of a DICOM conversion application that takes a        series of 2D DICOM images and a scan type as an input. The        conversion application utilises a series of image processing        operations that detect anatomical regions on each image through        automatic segmentation of the anatomy from background.    -   Thresholding procedure is used for segmentation.    -   One or more thresholds are applied to the 2D medical images in        order to automatically generate one or more different 3D        printable models allowing the user to select the printable model        with the least background noise. The generation of multiple        models is performed in parallel to increase performance.    -   The Web based viewer allows a user to easily select/annotate the        desired anatomy they wish to 3D print.    -   A preview of the anatomic feature for printing is accessible        before the order is completed.    -   DICOM images are processed in real time into a 3D model for        printing through a web application.    -   Data required to create a 3D printed model is uploaded along        with a prescription of specific needs.    -   The system displays an instant quotation.    -   Bespoke models are produced in 48 hours or less, which in turns        facilitates time constrained procedures, such as trauma.    -   A surface mesh is generated from the representation of the        voxels directly taken from the two dimensional DICOMS.    -   One or more end-users can have access to the Web Application.        The one or more end-users may be assigned different permission        or authorization levels.

1. Three Dimensional Printing in Healthcare Management Workflow—Insight

The workflow management software (called the Insight system) makes itpossible to routinely employ 3D printing for healthcare providers bymaking it simple to access the Axial3D tools. The software is availablethrough a web client and a command line API. The Web application allowsthe user to manage the process of procuring a 3D anatomical model forpre-operative planning and investigation. The Axial3D system alsoprovides reference client implementations (in python and javascript)that allow integration with the Axial3D server Application ProgramingInterface. The system is capable of receiving medical images fromstandard equipment. The system then stores those files to enable furtheranalysis and captures annotations to facilitate a prescription for thecreation of a bespoke 3D anatomical scale model printed using additivemanufacturing techniques. The system monitors the progress of the printthroughout the creation process. The system is therefore capable ofmanaging and reporting on the status of the print. Real time informationon specific timeframe before a print is ready is also available—takinginto consideration the segmentation, surface conditioning and printing.A 3D model can be produced in a matter of hours rather than days andweeks using existing technologies.

The workflow management process is novel in combining a number ofemerging technologies to provide a single service. We are usingBlockchain to provide secure and anonymous data transfer. Secondly weare leveraging the successes in machine learning as applied to imageprocessing and visualisation to create 3D objects from 2D images. Lastlywe are then taking these and preparing 3D printed versions of theanatomical objects. This is a unique combination of technologies thatwill deliver safe, speedy and secure 3D objects to clinicians in a timesensitive manner in order to allow pre-operative planning. These objectswill, in turns, improve the outcome for the patient being treated.

1.1 Upload—One Way Data Anonymisation

FIG. 7 is a diagram illustrating the workflow for uploading dataanonymously. Data can be uploaded via a reference web application or viacommand line tools to interact with the API. Data is then scrubbed toremove personally identifiable information from the original data. Inthis example, the end-user has a file for upload on their PC (71). Theend-user can view the files using their File System browser and selectthem for upload through their web browser. The files are anonymised forupload prior to leaving the user's PC. The anonymised files are thentransferred over the Internet to an application server, which saves themto the file system on a server (72).

1.1.1 Upload from PACS

A series of integrations, such as integrating directly with a PACSsupplier or a 3rd party client who manages secure exchange of data areprovided that allow end-users to send data from their PACS providerdirectly to the analysis platform to enable the printing of a 3D object.The analysis could be run as a service inside the PACS infrastructure.Or the data could be sent to the Axial3D server for analysis and a 3DPrinted object would then be returned. In the case of integrating withthe PACS supplier, the Axial3D server would be configured to act as ateleradiology site and receive data directly from the user's PACSsystem. Where the user has a 3rd party client for data exchange we wouldprovide an implementation of our client software to connect to theAxial3D server and facilitate upload and analysis of the data.

1.1.2 Secure Data Transfer and Handling through Blockchain

Blockchain technologies have in recent years risen to prominence as amethod for the secure tracking and sharing of information. They allow adecentralised and secure tracking system for any data interactions. Thearchitecture allows for storing proof of the existence of data withoutnecessitating the actual sharing of that data. This means it is possibleto prove the existence of data or metadata whilst maintainingconfidentiality.

For example, this could be the proof that a patient has given consentfor sharing of data without the need for us to see the agreement. Theinformation about the consent form and the patient data could all remainon the hospital system and they provide us with the proof that the dataexists. Therefore we can proceed with sharing without the need to seethe patient information. We only need to see the exact data required forprocessing.

We leverage this aspect of Blockchain technology for our data sharingplatform to ensure that we have received the correct permissions andordering information from users. We are using Blockchain to buildreliable clinical request contracts. Blockchain technology allows us tostand over the inviolability of the data we have received and to proofthe provenance of any data and associated metadata. This provides uswith the ability to track all events in chronological order. Once thedata is received onto the Blockchain it is not possible to alter it.This inviolability principle is crucial to providing assurance about theprovenance of information. This ensures that once the conditions for acontract are met it is possible to execute. Hence smart contractsexecuted on the Blockchain are used to order and/or initiate the 3Dprinting of medical data.

1.1.2.1 Data Initiation Phase

At the initiation phase, the data plan is agreed by all parties in asmart contract. This allows for the collection of information about therequirements of the prints such as for example: insurance status,payment status/limits, clinician, patient consent, annotations, datasharing agreements and/or data processing agreements. When data isentered into the Blockchain it is time stamped.

1.1.2.2 Print Initiation Phase

Before the print begins, more requirements are added to the 3dimensional file containing all relevant information for what isrequired for printing. Information about the stage quality gates isadded to the smart contract objects about the print. An example stagequality gate might be added by a quality control technician confirmingthat the physical printed model matches dimensional accuracy of twodimensional image it was created from. They would be able to addinformation about the protocol used to make this quality decision—i.e.the metadata about how the data was processed. Once the print object haspassed all the required quality gates in a smart contract it isavailable for printing. A smart contract can execute this stage.

We will receive an encrypted one way hash to initiate an order—This willallow the health care provider to initiate an order, without passingpersonal healthcare information to the Axial3D. This can be sent to theAxial3D server via the Blockchain and allows for the creation of a smartcontract. This contract can be updated with stage gate information asthe model is generated by Axial3D.

The Blockchain is also used to record the stage gates that the modelpasses through during processing, this allows us to automate theinitiation of a print once all the required steps have taken place. Thismeans that the network of distributed printers can be controlled via theBlockchain through the validation of a smart contract object in realtime.

1.1.3 PCoin

The Blockchain infrastructure facilitates the exchange of informationand data required to implement the printing of an anatomical model ofpatient data. Printing out a file also requires the utilisation of ourcurrency—PCoin. By linking the printing of an object to the PCoin weensure that the requirements of the smart contract are validated. Usersof the printer are motivated to ensure the validity of the Blockchainand the smart contracts as are those requesting the prints. This linkingof the physical product to the smart contract on the Blockchain allowsfor internally consistent pricing of the transactions and contracts.

Leveraging the Blockchain like this allows us to treat it as a safedeposit box, allowing handover of privileges. It is the ultimate audittrail for verifying that steps have been executed by parties in anagreement (smart contract). We enable the ability to encode theproperties of the data that is printed into a physical 3D printed objectand a virtual record stored on the Blockchain. This is facilitated by aQR code embedded in the 3D printed object and stored with the digitalrecord. This allows the linking of a QR code to a specific print. Theprocess of hardlinking is described in more detail below. Propertiesthat are recorded include the 3D representation of the printable objectand the information about the steps carried out by the Axial3D server tocreate this printable object. Therefore it is possible for anyone toverify that the appropriate quality control operations have been carriedout.

Print scheduling can be operated and managed using a DApp executingsmart contracts. This would be a Blockchain based application that sitsbetween the data processor and the printer detecting when prints areready to be sent to the printer. It is capable of identifying when thecriteria for printing have been met and of deciding how best to arrangethe printing on one or more printers.

Hardlinking allows for the linking of the original order, including thesmart contract and data (e.g. image scans and user specifications) tothe physical object used by the clinician and the auditing of all of themodifications of this data. Key to the print management process is thetransfer or realisation of the software defined print in the physicalworld. In order to track the transition a number of techniques areemployed to hardlink the software pipeline to the physical objectproduced on the printer. The aim of these techniques is to create anobject hyperlink. This hardlink is capable of being represented in anumber of different ways. It can be a simple computational hash of theoutput of the software pipeline. It can also be embedded into a QuickResponse (QR) code, NFC chip or RFID tag. This system allows us to movebetween the virtual world that created the 3D printed physical objectand the physical world that the object is instantiated in.

QR/Compliance code for tracking 3D printing is an inherently digitalprocess. We are generating a digital supply chain. Every part andprocess or modification is documentable and attributable. This createsopportunities for tampering and theft. We have the ability to trace theobjects from order initiation through to production and potentially tousage/implantation. Through hardlinking a single object is provided fromconception to implantation. Hardlinking is the registration of a uniquecode to a single print—covering the entire pipeline from initiation toprint. This system provides the infrastructure and oversight necessaryfor the management of a 3D printing facility to provide anatomical scalemodels from medical images. The system allows users to upload medicalimages directly and annotate those images in the workflow. We provide areference implementation via a website and a command line tool thataccesses our application programing interface.

At the initiation of an order a cryptographic hash of the patientinformation is created by the ordering organisation. This allows them toidentify a patient without passing over any sensitive patientidentifying information. This is crucial for the system as it allows theorganisation placing the order to validate for itself that the modelprinted is associated with the correct patient without disclosing thepatient data.

1.2 Print Management

Tools are provided in order to determine if a given volume is printableand to identify problems with the printability of the object. Based onan understanding of printing volumes rendered by computational or manualtechniques, the printability of 3D physical model of a patient specificanatomic feature has been extensively studied and understood, such as:how to print a diverse set of objects within a scene, the management ofprinting a diverse set of objects, including the ordering, orientation,placement on plate and across multiple types of printer and printingtechnologies.

Connected machines may also operate in conjunction with each other tocreate a required 3D print. This requires a coordination layer sittingbetween the image processing algorithm and the printer to determine theoptimal print delivery pipeline.

Text messaging may be used to report on status and initiate commands tothe server.

1.2.1 Scheduling of prints Print scheduling is used to facilitate thevolume and timing of print. Timelines for prints are scheduled based onprints finishing within a 10 hour working window and based on inboundmodels and surgical requirement. Delivery schedules are also based onachieving lowest cost for arrival of an on time model—linked with thecost of the postal service and provider, as well as method oftransportation, including autonomous vehicles known as drones. In eachcase, an equation can also look for models that can fit within its printvolume for a single print.

Some examples below are given, in order to meet the required date N forthe delivery of a 3D printing model:

Model required date=N

i.e. If N>72 hours from upload—lookup—N<72 days if no prints andschedule 48 hour royal mail pick up if UK—if EU US shedule 48 hour DHLUS & EU;

if N<72 hours lookup—instances for N<48 hours—if no prints and schedule24 hour royal mail pick up if UK—if EU US schedule 24 hour DHL US & EU;

if N<48 hours lookup—instances for N<24 hours next available printerslot and schedule—if no prints and schedule 24 hour royal mail pick upif UK—if EU US schedule 24 hour DHL US & EU;

if N<24 hours lookup—next available printer slot and schedule drone pickup=length of print time+120 minutes;

If N<24 hours and not within drone/unmanned vehicle range complete ifstatements above to the closest geographical location of print farm todelivery address of order.

1.2.2 Automated Material Selection

Different parts of the anatomy require different approaches to 3Dprinting. The printing of multiple pieces of anatomy in a single printor combining them requires the construction of bespoke printingstrategies.

To meet the expectations of users of the 3D prints, the selection of thematerials used are optimised to print the anatomical features for theirresemblance to the material used for printing. For instance hard bone isprinted in harder materials and soft tissue in softer materials. Hence,the software automatically selects the material for printing. Thisselection may also be based on the requirements of the model, e.g. speedwith which it is required.

There are restrictions on the possible size of the prints using existing3D printing machines. To overcome this for oversized pieces of anatomywe construct custom features that allow the combining of multiple 3Dprints into a single 3D printed model. The placement of these connectivepieces follow a set of rules that which have been formalised such thatthey can be applied also to new models. To achieve this, the Axial3Dsystem is able to predict the type of material required for theseconnective pieces.

1.2.3 Automated Printer Selection

Depending on the type of anatomy to be printed we will select thematerial and technology pairing best suited to the final model. Thisrelies on the ability to clearly identify the anatomy beforehand. Aninput to this process is a pre classified piece of anatomy (taken fromthe medical scans) which is present in the object or objects to beprinted. The output of the process is a selection of the specificprinter that is best suited to the anatomy being printed. Printers havethe following dependencies: build volume, materials available, minimumfeature size. The parameters involved in the selection of the 3D printerare: volume of 3D model, hollow structures within the model & minimumfeature size. The algorithm matches the printer parameter to the modelparameter as described above.

We have built up a series of profiles of users and their preferences forparticular model types. For example we have data that indicates thatoncology users prefer prints that can incorporate clear models. We willuse this information to predict what kind of technology the model shouldbe printed on for the user's requirements.

1.3 axial3D Automated Pipeline

FIG. 8 shows a diagram illustrating the Axial3D Automated Pipelineworkflow; once the data is received (8.1), the Axial3D automatedsegmentation will identify the anatomical components within the scan and“label” them accordingly (8.2). The Tissue and Organ Segmentation isthen used by the surface generation in order to create the organ volumes(8.3); the volumes are then passed to the conditioning phase (8.4) whichwill identify the volumes that require printing, the associated correctmaterials and perform the printer selection. The three objects producedby the Axial3D Automated Pipeline stages are used in different ways: thesegmentation data and the 3D Mesh produced by the Segmentation and theSurface Generation stages (8.5 and 8.6 respectively) are used in theAxial3D review component of the Web app; the user can interact with themby adding annotation, select components to be printed and selectpreferred materials for specific anatomical features; the 3D printableobject (8.7) is used at print time. All output files and data are storedin the Axial3D Data Storage and can be downloaded using the Axial3D CLI.All steps are described in greater details below.

1.4 Model Annotation

The user is presented with the 3D model that is generated using thesteps described in the automated pipeline. This process identifiespieces of the anatomy visible in the scene. This allows the user tointeract with the 3D model on screen. Once the model is shown on thescreen the user can zoom in and out to visualize the model as well asprovide annotations. These are free hand drawings that are mapped ontothe model by our software. It is possible for this annotation process tobe completed on a mobile as well as on a desktop or on other devices.

The annotations provided by the customer are then used to, for example:

-   -   1. clearly and accurately define regions of the scene that they        want to print;    -   2. provide information about the anatomical components contained        within the scan, facilitating the task of anatomical        identification (see below).

1.5 Image Manipulation and Download

Protocols for the handling of data by expert medical visualizationengineers have been developed. In addition, tools for the exchange andstorage of data and annotations generated by the healthcareprofessionals have been developed.

It is possible in this view to overlay the 3D models, and the originalscans as well as the automatic segmentation results in order tofacilitate validation of the segmentation algorithm. Through the webapplication the results of the automated segmentation are presented andthe engineers are able to approve the segmentation. Tools and aninterface also allow the modification of the segmentation within the WebApp.

Methods are also developed to increase the speed of transfer of dataabout medical images including specialized formats and data structures.This includes the identification of duplicate information prior to datatransfer, enabling only the sending and receiving of unique information.Hence less information needs to be sent, resulting in faster transfertimes. This is key to ensuring that users experience faster loadingtimes and a better user experience. This requires the development ofinfrastructure that enables the storage of a large amount of dataroutinely. This also requires significant investment in security toensure that patient data can be handled safely and securely.

Post image analysis, it is possible to download annotated medical imagefiles. These contain the output of our automated segmentationalgorithms. They can be viewed in a standard medical image viewer. Theseimages are copies of the original.

1.6 User and Task Management

FIG. 9 is a diagram illustrating the main aspects of the workflow fromthe surgeon initiating an order to the delivery and utilization of the3D printed model by the clinician or healthcare professional. Theprocess of ordering and generating a 3D printable file and subsequentprinting of it can be broken down into a series of one or more tasks.Each of these tasks can be carried out by a person or automated. Weprovide the infrastructure, services and software as well as managementinterface to facilitate the implementation of these tasks. We offer atask-based approach to executing the process of creating a 3D anatomicalmodel. We allow multiple users from an organisation to participate inthe process. This is in addition to the use case of a single useroperating the entire workflow on their own. This encompasses imageprocessing specialists who can interact with the data upload stages.Medical professionals can upload data about the model being created andannotate the model as it progresses through the workflow, as shown inFIG. 9.

We support all aspects of the ordering process within a healthcaresetting. Our software allows the role based provisioning of the tasksrequired to order a 3D print. From the surgeon requesting the print,anonymisation of the data uploaded by a radiologist, sign off on thespend by clinical director, procurement decisions by the healthcareprovider, transfer of files from the imaging systems, management of the3D print process internally and tracking all these authorisations withinthe system. The tracking facilitates the production of an audit trailcapable of providing the required information about the transition ofthe data through the workflow into the final 3D printable object. Theworkflow management software holds all user types and assigns rolesappropriate to the user controlled by the group administrator. Users canbe grouped together for example by membership of the same organisationor group of organisations. In this case an administrative user must becreated to manage roles and access.

1.7 Improved Healthcare Experience Through 3D Printing

Patients experience an overall improvement of the level of serviceprovided by the healthcare institution when 3D printed models are used.The models can be used by the medical staff to improve thecommunications of the conditions to the patient, allow a more informeddecision as to the course of treatment to be followed as well as improvetheir confidence on the medical staff and institution caring for them.This will lead to increased buy-in by the patient thereby.

2. Axial3D Automated Segmentation

The goal of image segmentation is to assign to each pixel within adigital image (volume) a label corresponding to a given class (bone,fatty tissue, tubular tissues such as veins and arteries, tissue withcavities such as lungs, etc) of possible objects that may appear withinthe image. The classes may represent any kind of object that may appearwithin the image, including human tissues, organs and foreign objects(the scanner bed, metal implants, pacemakers etc). This means that wetake medical images as input and produce a new volume superimposed ontothe original scanned volume where each pixel of the original volume isreplaced by the label corresponding to the appropriate tissue type.

This section will outline the algorithms implemented by the Axial3Dserver in order to achieve accurate and reliable segmentation of themedical scan for the purpose of 3D printing. Each algorithm has a numberof advantage and disadvantages. The segmentation pipeline includes afinal step where the results of all algorithms are combined together toderive a final segmentation that is then used to combine the results ofthe various algorithms.

FIG. 10 shows the Automated 3D Segmentation workflow. The input dataconsists of the scan DICOM stack provided by the customer; the stackcontains the 2D images obtained from the medical scan; automatic anomalydetection will flag the presence of gross anomalies and inconsistenciesin the input data; the data is then passed to any number of segmentationalgorithm (three shown in this example), each algorithm is runindependently from the others; each algorithm produces a segmentationmap; the results of all Segmentation workers are then combined by theReducer; the final Label DICOM stack is then outputted and passed to the3D surface generator code.

2.1 Anomaly Detection

Anomalies and erroneous information are often contained within the datathat is received with the patient order. Such anomaly present itself asa mismatch between the expected form of the data and what is beingreceived; this mismatch may appear both as a gross or very subtledeviations from the expectation. Anomalies are any form of deviationfrom the expected appearance of a scan that would impair the results ofany of the automated algorithms and lead to incorrect 3D models.

Possible sources of anomalies are:

-   1. human errors such as: scan data may be classified as a computer    tomography where in reality it is an MRI scan or incorrect patient    data;-   2. presence in the scan of foreign objects that causes aberration    and errors in the scan data;-   3. low quality scan images dues to issues with the scanning hardware    or inconsistencies in the scanning protocol (wrong X-Ray energy and    flux; wrong scanning settings used during the scan; worn out X-Ray    source and/or other equipment);

The automated pipeline will detect these anomalies by comparing theinformation provided against a simplified standard set of parametersexpected within the 3D volumes; this standard implements a number ofsimple rules on the expected color histogram to check against: forexample, CT scans will generally display strong components in the regionof −1000, −80, 80 Hounsfield units roughly corresponding to fattytissues, tissues containing water and air; the lack of all these threecomponents will indicate that the scan may not be CT as specified by theuser. The same applies to MRI scans. Detection of such cases will resultin marking the scan for revision by an engineer in the workflow.

2.2 Threshold

The simplest method of image segmentation is to use a thresholdingalgorithm based on the knowledge of the typical Hounsfield of bones.FIG. 11 is an example of medical image segmentation using a thresholdestimate (see below for an outline of the automated threshold estimationalgorithm). In this approach, all pixels with a value above 193 areclassified as bone, while pixels below this value are non-bone.

The Axial3D threshold algorithm expands on this concept by using alogistic probabilistic function instead of a hard threshold in order tocalculate the likelihood of a pixel of being the tissue in questioninstead of a binary value. e.g. instead of “bone” VS “no-bone”, thethreshold calculates a value between 0 and 1 where higher numbercorresponds to likelihood of being bone).

The main issues with this method are:

-   1. Noise of non-bone tissue: it is often found that non-bone tissues    may light up and appear as small regions with pixels with values    above 190; this is often caused by the limitation of the imaging    technology (the 2D tomographic reconstruction in particular) which    may create small artefacts;-   2. holes in bones: for the same reasons discussed above, it is very    common for small regions (typically 5-10 pixels in size) of bone to    appear dimmer within the image (less than 190); these regions appear    as holes in the bone;-   3. foreign objects: it is quite common to observe foreign objects    within the CT scan such as the bed onto which the patient is laid;    these object often have very high Hounsfield Units (HU) due to their    relatively high density; these objects are often segmented as bone    due to their large size and high pixel value.

The issues described above are partially mitigated using a smoothingalgorithm (Gaussian or median filtering); such methods improve theoverall quality of the segmentation however will slightly reduce thefidelity of the bone volumes.

Thresholding in combination with a small pre-processing filtering(Gaussian) has been found to be the most appropriate method of imageanalysis for the generation of Web-App models. The models haveacceptable anatomical fidelity as well as a very low number ofmislabelled bones as a relatively small number of holes. All anatomicalfeatures are present, with the exception of very small bones such ascarpal feet and hand bones, in particular in proximity to the physic andmaxillofacial features in children and in elderly people (in particularsubjects with advanced arthritis).

2.3 Programmatic Threshold

We have demonstrated that it is beneficial to apply different parametersas inputs to our algorithms on the basis of the application area.Specifically, we have optimised the creation of the model based on keyparameters such as the scan type, bone type, tissue type, age, genderand weight of the patient. We capture this information from the imagesand by capturing this in the user interface to our software.

The Scan Type and Bone Type of the DICOM image series are key parametersthat provide information that is used to make the threshold process morerobust and accurate. The end user selects a Scan Type (ST) of CT, MRI,PET or SPECT and a bone type (BT) of Hard or Soft. The threshold stepsegments out the bone from background information e.g. skin and foreignobjects such as a bed. It is critical that all of the bone (and only thebone) is segmented during this step. Failure do this will generate aninvalid 3D model.

A considerable amount of work has been carried out in order to determinethe best threshold value to be used for a given CT scan. It has beenfound that the ideal threshold varies depending on the CT scanningparameters (X-Ray energy and flux); these settings are ordinarilymodified by the radiologist performing the scan in order to maximise thequality of the scan.

In order to obtain the best segmentation results, the Axial3D AutomaticThreshold Estimator uses an analysis of the scan color histograms inorder to derive the best estimate. The algorithm performs the followingsteps:

-   -   1. detects the peaks of the histogram corresponding to the        tissues that are similar in color compared to the tissue of        interest;    -   2. once the peaks are detected, their inflection points are        derived by calculating the zero of the histogram second        derivative in proximity of each peak    -   3. the offset between the peak and the inflection point is        derived;    -   4. the estimated threshold corresponds to the position of the        peak with an offset corresponding to three time the inflection        offset;    -   5. if an error occurred during the peak detection, a default        threshold is used depending on the tissue of interest and scan        type; errors include the lack of peaks in direct proximity to        the tissue of interest, peaks may not be sufficiently prominent        (too small); the scan modality and/or tissue type may not be        supported by the algorithm

FIG. 12 shows the bone threshold estimated using our method for theHounsfield unit histogram from 24 CT scans. As it can be seen, the 24histograms vary considerably; this variability is due to the varyingratio of different tissue type within each scan as well as the CTscanner settings (voltage, flux etc). FIG. 13 shows the histogram of thethreshold across 175 CT scans. The histogram shows the variability ofthe estimated threshold; for the majority of scans the estimatedthreshold is around 100 Hounsfield unit; this is the lowest thresholdvalue that can be used in order to remove all non-bone tissue from thesegmentation results.

2.4 Decision Forests

Decision trees are an improvements over the threshold method describedabove. They involve the creation of a “tree of thresholds” where pixelsare classified based on a number of properties, not just their value.Any property of the pixel as well as its neighbours can be used tocreate a new decision.

In the context of our application, the following properties of thepixels have been selected in order to create the decision tree:

-   1. how many pixels looking almost like bone are near the pixel in    question;-   2. how many pixels looking exactly like bone are near the pixel in    question;-   3. how strong is the overall gradient of the image at the given    pixel if the consistency of the gradient direction within a small    neighbourhood of the pixel; Sobel filters are used in order to    derive this property;

The decision trees are trained using existing pre-labelled Axial3Dpatient data. The decision tree is applied to a subset of pixel withinthe original scan; the labels obtained from this subset are then upscaled using standard interpolation methods in order to recover thesegmentation of the full image. The pixel subset is generated bysubsampling the original images, typical subsampling strides are 5, 7 or13 in both the X and Y direction. The subsampling stride is selecteddepending on the pixel size (derived from the DICOM themselves); thesubsampling stride is selected so that the subsampled resolution doesnot fall below 5 mm in the original space.

FIG. 14 is an example of medical image segmentation using a randomforest. Decision Trees produce lower fidelity segmentation compared tothresholding algorithm(s) (the edges of the anatomical components arenot as accurate), however it produces less noise.

2.5 Chained Decision Forests

In order to overcome the poor resolution of a single decision treeforest approach, a new approach based on a hierarchy of decision forestsis developed.

FIG. 15 is a diagram illustrating the general concept. The results fromthe decision tree and the results from another classifier (e. g. thethreshold) are fed to a new decision tree alongside the original pixelvalues. In this approach, each forest-node is treated as a simpleclassifier that produce a score as to how likely the pixel is to beingpart of a bone. Each following tree will consume this estimation andmake an overall assessment of the likelihood of the pixel actually beingpart of a bone.

2.6 Neural Network

Neural networks are similar to the Decisions Forests described above inthe sense that they are built by creating very simple decision units andchaining these units together, so that what has been discovered by a fewunits helps a following unit to make a new estimation as to what thenetwork is looking at.

Example of an advantage and a disadvantage of using Neural Networkmethods are the following:

-   1. Advantage: as each neuron can be trained on recognising one    feature which may be either present or absent from the image, Neural    Networks are extremely good at generalising common features of the    subject and outperforms almost any other algorithm in the detection    accuracy (including decision trees);-   2. Disadvantage: although some neurons may be “looking” at a small    patch of the image, any following neuron will be looking at larger    and larger patches of the image; in most Neural Networks, the final    neurons will be classifying very large sections if not the entire    image, which may lead to very poor resolution of the boundary of the    detected object.

2.6.1 Fully Convolutional Neural Networks: FCNN

In a typical Neural Network, the final layer will produce estimations ofthe input pixel belonging to a given class; in the most common approach,the neural network will take a full image of a fixed size as input andclassify the entire image as to belonging to a given class. Thiseffectively means that in the standard approach, a neural network doesnot produce a segmentation of the image, but a classification of theentire image; this is a little like answering the question what objectis in the image, not where the object is.

An approach to solve this issue is to use a specific layout of neuralnetwork called a Fully Convolutional Neural Net. In this approach, alllayers of the network are convolutional, including the final layer.Using this approach allows the system to minimise the down-scaling ofthe network using max pooling.

The final output of a FCNN is typically a segmentation array with areduced resolution (a reduction of 16*16 is typical for this kind ofapplication when using 4 2*2 max pooling layers). Just as for thedecision trees discussed above, this method is expected to produce veryconsistent results, however a method for dealing the downscaledresolution is necessary.

2.6.2 UNETs

One of the key goals of a UNET neural network is to upscale the resultsof a typical neural net to the original size of the image and obtainaccurate boundaries of the detected object. This is done by adding anumber of up-scaling layers where the outputs of previous layers areused to identify the regions of the image and lead to a specificclassification. Due to their reliance on generalised features, neuralnetwork and UNETs in particular are used at Axial3D in order to generateprint ready models of CT bone tissue, bone type classification(trabecular, cortical, etc) as well as MRI scans segmentation.

3. Anatomical Feature Identification

3.1 Axial3D Atlas

In order to facilitate the image segmentation of medical scans and toperform further classification of the detected objects, an existingknowledge of the anatomical features that may be present and theirappearance within a diagnostic scan (CT, MRI, etc) must be available.

Axial3D uses a graph database in order to store such information in anATLAS of human anatomy relevant to medical scanning techniques. In sucha system, each anatomical feature is represented as a node andrelationship between anatomical components are represented asrelationships. Both nodes and relationships between them can be of anykind such as simple proximity, attachment, ligament, functional, etc;the nodes can represent different tissue types and organs. Both nodesand relationships contain additional information; in particular theycontain a reference to a image containing such anatomical feature, itssegmentation, general features of the anatomical object such as volume,surface area (if applicable), average and Standard Deviation of itsHounsfield Units.

Key features of ATLAS are:

-   1. to provide the Axial3D R&D and segmentation group a simple and    fast information retrieval system (accessing the relevant scan and    segmentation data);-   2. provide a centralised area where specific information on    anatomical features is stored (organ appearance and properties);-   3. provide a simple method to access information of related organs;-   4. provide a backbone of Ground Truth data for organ classification    and automatic medical scan interpretation.

Typical examples of ATLAS usage include, but are not limited to:

1. retrieve the reference of all scans in our data-store related to hipsin CT scans;

2. determine the volume of all radius and ulna available in the scans;

3. determine the histogram of the Hounsfield value of lungs;

4. find all the bones typically located in close proximity to theClavicola;

FIG. 16 shows a sample of a graph database showing the primary featuresto be included: the nodes represent conceptual features of the humananatomy such as regions of the skeletal system and specific bones; allthe nodes are connected between each other by a belonging relationship(the ulna belong to the limb which in turns belong to the human body).

3.2 Standard Anatomical Model

A set of standard anatomical models are generated by retrieving all theavailable data within our data-store for a given anatomical feature inits healthy state (no pathology). The available segmentation data for agiven anatomical feature (a bone, a specific organ) is registered andscaled in order to align the segmentation from the different data-setscorresponding to the various scans. The aligned and scaled segmentationdata is then used to extract and compare the Hounsfield values as wellas the CT scans and/or the MRI value; the similarity between each scancan then be used to generate the standard anatomical model of thefeature and its expected appearance in a medical scan. Such models mayalso need to take into account some important aspects related to thepatient from which the data comes from: for example, anatomical featuresare expected to vary over time as the patient ages (grow larger, someorgans may become “worn out” as a result of arthritis on bones forexamples, etc), sex and clinical history (some pathology will affect theappearance of the scans).

The Standard anatomical model is stored as a reference within the ATLASand ATLAS is also available in order to facilitate the retrieval ofdata. As new data is added to the Axial3D database, the standard modelsare updated to include the new data; the history of standard modelgenerated for each anatomical feature is also preserved.

3.3 Interesting Feature Extraction

ATLAS is used to store and retrieve data related to “interestingfeatures” of each anatomical component. In a similar fashion as for theStandard Anatomical Model, all the segmentation related to a specificanatomical component of the human body can be retrieved and featureextraction algorithms are applied. The feature extraction algorithmstake advantage of both the segmentation as well as the actual scan datain order to obtain interesting properties of the tissue or organ underconsideration.

Some simple feature extraction algorithms include:

1. the anatomical component volume,2. the component surface area,3. its average Hounsfield unit (if CT) and standard deviation across allavailable scans,4. histogram of the Hounsfield Units across all available scans,5. Smallest bounding box that contain the anatomical component.

More complex features may include, for example:

-   1. key-point detection: determine the presence of keypoint    landmarks; standard algorithms can be used for this purposes such as    Fast, SURF, SIFT, ORB, etc; these algorithms will require adaptation    in order to work with three dimensional data; the detected keypoints    can be compared between the various scans and similarity between    them can be preserved in the ATLAS data-store;-   2. Structural and shape analysis: a number of predefined shapes and    volumes can be detected within the anatomical feature being    considered: for example, within the femur, the main body can be    compared to a cylinder and the two heads as two spheres;-   3. Anatomical Landmarks: in this approach, specific features that    are unique to the specific anatomical component are detected.

The extracted interesting features are then added to the Axial3D ATLASas part of the node properties to facilitate future reference. Axial3Dis in a unique position to create such a database: due to the largevolume of CT scan data as well as the availability of high qualitysegmentation for the scans, Axial3D can generate values that are bothstatistically relevant and sufficiently general to allow the creation ofmore advanced detection and classification algorithms (see below).

3.4 Anatomical Component Classification

The standard anatomical model and the extracted interesting features canbe used in order to derive a reliable and consistent classification ofthe anatomical components located within the scan.

The general approach to perform such a task is the following:

-   1. derive accurate segmentation using the automated segmentation    algorithms;-   2. apply the feature extraction algorithms to the segmentation in    order to derive the values of such features; note that these    features may be erroneously detected if the segmentation produced    poor results;-   3. compare to the existing data-set of interesting features and    attempt to find a number of matches;-   4. the matches are constrained and filtered depending on the    proximity map derived from Axial3D ATLAS;-   5. the standard models are used to further refine the filtering and    cross-checking by fitting a linear transform between the    semi-classified segmented objects and what the standard model looks    like;-   6. due to the inherit inaccuracies of the segmentation step, each    refinement of the matches produce a score or probability of having    matched the anatomical features correctly;-   7. the set of scores obtained can be used in a decision tree (or    forest) in order to derive the final classification of bones;

Note that this method will not identify all anatomical features presentwithin the scan; it will however identify a sufficient number of them touniquely identify the region of the human body that has been scanned,its orientation, as well as the classification confidence.

3.5 Segmentation Curing, Deformities and Pathology Detection

The Anatomical classification described above provides the basis todetect any deviation from standard of a given scan:

-   1. Touching organ curing: It is often found that several anatomical    components made up of the same tissue may be located in close    proximity to each other; this is the case for example of tarsal    bones, carpal bones and other joints, vessels and other cardiac    tissue, etc. In some instances, the tissue that makes up the bulk of    the organs are all segmented correctly, however, due to their    proximity, the organs themselves are not separated correctly as    individual components; in such scenarios, the touching organs will    be detected as a deviation from the standard appearance of the scan    and a specific algorithm for edge finding can be used to separate    them as individual entities; once the segmentation curing is    performed, the Anatomical Component Classification can be    re-estimated and the classification confidence should have increased    considerably (as more anatomical features should now be matched to    the Axial3D ATLAS);-   2. Implants Artefacts curing: it is quite common to obtain error in    the segmentation due to the presence of foreign objects that have    been implanted within the patient (pacemakers, tooth filling and    braces, nails to cure scoliosis, etc); these object will appear    within the scan as foreign objects with completely different    properties compared to any other tissue, in particular when made of    metal (titanium implants, etc); these objects will inevitably cause    errors in the segmentation algorithm by introducing holes as well as    misclassified organs; in a similar fashion as the touching bones    algorithms described above, the implant can be identified as such    with a separate tailored segmentation and classification set of    algorithms; the identification of such items can then be used to    correct potential errors in the organ segmentation;-   3. Deformities and Pathology Detection: once the components and the    scan region is identified, deviation from the normal appearance can    also be used to assess the presence of some form of pathological    condition and deformities; such estimation is different compared to    the standard pathology detection that is performed using neural nets    to classify/segment the patient data directly, it rather employs a    comparative analysis of the expected appearance of an object and    derives deviation from such appearance; examples of applications of    such methods are the determination of skeletal deformities (flat    foot, scoliosis, etc), traumatic fractures, etc.

3.6 Reconstruction Using Comparison

FIG. 17 shows the results of image reconstruction using comparison.Boolean differences for bio printing can be used to assess the volume ofmaterial from a statistical model of pre-classified ‘healthy’ volumes oftissues and create an automatically generated 3D file of missing volume.Our automatically segmented data is assessed against statistical modelof pre-segmented anatomy ‘Best fit model’, is created based on astatistical model for patients' anatomy to show optimal reconstructionof tissue. Missing fragments are predetermined with a best fit model andtissue scaffold models created from this.

4. 3D Model Creation

In order to obtain a 3D printable model, the segmentation data generatedby applying the image segmentation algorithms, anatomical componentidentification and segmentation correction, a 3D surface mesh must beextracted from the scalar volumes.

For a 3D mesh to be printable, it must have the following properties:

-   1. all disjointed surfaces are closed manifolds (3D volumes);-   2. appropriate supports are necessary to keep the disjointed    surfaces/volumes in place-   3. appropriate supports are necessary in order to facilitate 3D    printing;-   4. in order to be printable using SLA technology, all surface    volumes must not be hollow (no close surfaces fully contained within    other surfaces);-   5. if a hollow volume is specifically requested by the clinical    staff, appropriate drainage holes must be added manually by the    operation team.

Once the 3D models are generated, they must be reviewed and approved bya biomedical 3D printing technician within the operations team. Whenevernecessary, appropriate modifications are made by the operations teamprior to printing.

4.1 Surface Extraction

For each scan, a 3D of model (surface) is created from the segmentationobtained using the methods described above. Different methods areavailable including: marching cubes, marching tetrahedrons, surface netsand other isosurface methods.

A considerable amount of work has been done to compare the results ofthe algorithms in order to obtain the best results in terms of thefollowing aspects:

-   1. anatomical accuracy: the resulting surface must be as close to    the original volumetric data as possible;-   2. Minimise the cuberille effect: since the iso-surface is derived    from the labelled data (the segmentation) it is important to    minimise the block appearance of the resulting mesh; this can be    achieved by modifying the marching cube algorithm in order to force    some vertices/faces to be placed on voxels that do not intersect the    iso-surface directly;-   3. 3D printing specific requirements: the mesh should be as close to    a printable model as possible in order to facilitate the following    surface conditioning steps; this means that the resulting mesh must    be a manifold.

4.2 Surface Conditioning

Iso-surface extraction has a number of potential issues that need to beaddressed by a conditioning stage:

-   1. High Poly Mesh: meshes obtained using any iso-surface extraction    algorithm tend to generate models with a number of faces that is    proportional to the number of voxels in the input volumetric data;    meshes obtained using these approaches tend to be extremely high in    their number of polygons; such a level of details is often    unnecessary (a large planar surface can be represented by one single    large polygon instead of several hundred small segments);    poly-reduction algorithms are applied to the meshes; example of    these algorithms include short edge collapse, quadratic edge    decimation;-   2. Mesh errors: this includes errors such as duplicated points,    overlapping surfaces, missing surface; removing point-like surfaces    (small unnecessary surfaces); the goal of this step is to correct    some simple errors and ensure the mesh is a manifold;-   3. Mesh filtering: once the meshes are reduced in poly number to an    acceptable level and mesh errors are removed, simple filtering    algorithms are applied in order to minimise the “pixelisation” given    by the original segmentation data; the filtering radius is reduced    to a minimum in order to preserve anatomical accuracy.-   4. Cover holes: some organs are often only partially visible within    the scan slices; in these situations the surface mesh obtained from    the segmentation will not be closed; in these situations the holes    are covered by additional triangles; this step must be performed    after the mesh error correction and produces best results when    applied after the smoothing as well; the goal of this step is to    ensure all surfaces are closed manifolds.-   5. Textures: The meshes are displayed to the user. We will select    appropriate textures that allow the user to better visualise the    anatomy. This will mean the automatic selection of different    textures and lighting procedures within the display view of the    application.

The expected results of this stage of mesh processing is to obtainanatomically accurate and good quality meshes that need minimaladditional processing to make them “print ready”. The resultingconditioned meshes are also used for displaying purposes within the webapplication after a further poly reduction (typically to approximately10,000 faces).

4.3 Print Ready Models

The conditioned meshes need a further processing step in order to beprintable. This step ensures that the 3D printing will successfullycomplete with no failures. This step includes the followingmodifications to the anatomically accurate models:

-   1. Fill in the watertight surfaces: some organs such as bones, blood    vessels, stomach, etc are naturally hollow in shape resulting in a    watertight hollow surface; these surfaces may fail to print when    using a SLA printer due to the lack of drainage channels for the    trapped resin; these organs may need to be filled prior to printing    (specific requirements from the clinical staff may specify    otherwise);-   2. Addition of dowels to support the various organs: the 3D printed    organs will require supports in order to keep their position; small    dowels are added along the shortest line to join the organ together;    once all possible dowels are added between all organs to be printed,    a specific algorithm will remove the redundant dowels starting from    the longest;-   3. Addition of print supports: in most 3D printing technology the    addition of small supports is required in order to facilitate the    printing and achieve the best results; the supports are added by    determining all local minima of the surface; the print supports are    removed during printing post processing.

5. Post-Processing of 3D Printed Models

Post-Processing techniques—clear contrast in sterolithograpghy models.

Models are created based on solid exterior X of a model and a number ofinternal points of interest Yn. A boolean difference is completed on Xby removing Yn from its internal structure automatically. A hollowcylinder of for example 3 mm diameter is created from the surface of Xto the surface of Yn with the shortest distance taken.

Pre cut model construction—Models that are outside the bounding box ofthe available print volume of the printer being used will be analysedand the bounding box −5% of total model volume will be taken and a cuton the ZY plane will be automatically created. Both models will have4×3×3 mm cylinder holes inserted in the largest surface area of the cutface. 4×2.8 mm×2.8 mm cylinders will also be created and added to theprint file.

Model created in situ for bone placement analysis—bones within the scenewith pre-segmented anatomy will be assessed and articular surfaces willbe detected and cylinders will be inserted between two central points ofarticulation. Diameter of cylinder will be determined on 50% ofarticular surface cross sectional area.

6. Denture-Store

The methods and systems described above can also be applied to generate3D printed physical models for dental and orthodontic labs.

In typical traditional orthodontists' practices, impressions are madeand stored for up to 5 years (patient records). In busy practices, thisends up being a huge amount of physical impressions having to be stored.

Dental impressions from across the years can be 3D scanned (usingavailable 3D scanners or with an IR laser and a turntable) andcatalogued on the site.

A white light oral scanner may be purchased and all impressions may beelectronically done, stored and sorted in a program.

All impressions, legacy or new, can be sent to the Axial3D system inorder to generate a 3D printed physical model. The printing of the 3Dphysical model can also be directly completed within the dental ororthodontic lab. So all that is required is a computer and a printer tostore patient data within each lab.

7. Phantom Production

Phantoms may be created for use in X-ray and CT/MRi calibration.

Different 3D printer filaments, resin or liquid have been developed

A material that will not absorb the solute containing a radioactiveisotope is also developed; i.e. the model can be washed with water afteruse and be non-radioactive.

8. Axial3D training:

Modular system that allows surgeons to train on various surgicalprocedures without the requirements for cadaver models. A pre-segmentedvolumetric representation of a human is displayed, using for exampleaugmented reality (derived from both CT & MRI data) and a surgeon isable to swap out healthy tissues with the ones from Axial3D's databaseof rare pathologies in either virtual or physical 3D printed format.Haptic feedback is incorporated into the application to mimic differenttissues when carrying out procedures.

APPENDIX: CONCEPTS SUMMARY

This section summarises the most important high-level features describedabove; an implementation of the invention may include one or more ofthese high-level features, or one or more of the key subsidiaryfeatures, or any combination of any of these.

Concept A: Entire Workflow for Generating 3D Printed Model of a PatientSpecific Anatomic Feature from 2D Medical Images—Performed Automatically

A method for generating a 3D physical model of a patient specificanatomic feature from 2D medical images, in which:

-   -   (a) the 2D medical images are uploaded by an end-user via a Web        Application and sent to a server;    -   (b) the server processes the 2D medical images and automatically        generates a 3D printable model of a patient specific anatomic        feature from the 2D medical images using a segmentation        technique; and    -   (c) the 3D printable model is 3D printed as a 3D physical model        such that it represents a 1:1 scale of the patient specific        anatomic feature.

The method may further include the following optional steps:

-   -   (a) Clinical staff can upload the 2D medical images and any        additional information to the Axial3D servers using the Axial3D        online web application;    -   (b) Clinical staff can add annotation and patient specific        prescription through the Axial3D web application in order to        personalise the final 3D printed product and obtain the desired        results;    -   (c) Clinical staff can interact with any aspect of the automated        3D model generation in order to improve the final models,        personalise the results and flag occasionals problems with the        models prior to production and shipping;    -   (d) the Axial3D automated 3D model generation is a fully        automated system that converts the 2D images into a final 3D        printable model; the users can in real time review, annotate and        modify such models through the web application prior to        production and shipping;    -   (e) 3D model of the anatomical feature of interest (skeletal        structure or joint, organ, specific tissue, etc) created by the        automated system and approved by the customer is 3D printed in a        1:1 scale;    -   (f) The Axial3D web application employs the Axial3D fully        automatic image segmentation and anatomical recognition pipeline        in order to achieve near real-time display of the 3D models and        allow seamless and near instantaneous update of the 3D models        based on the customer feedback;    -   (g) All communication and the transferral of data through the        network and the web application is anonymized, secured using        encryption and takes advantage of Blockchain in order to        orchestrate the workflow, minimise the risk of loss of data,        increase transparency and minimise the time required to deliver        the final 3D model;

Optional features (each of which can be combined with others) includethe following:

-   -   2D medical images are uploaded alongside metadata.    -   metadata includes: patient's prescriptive information, medical        professional information, patient information;    -   2D medical images are anonymised prior to being sent to a        server;    -   the patient specific anatomic feature is automatically        identified from an analysis of the 2D medical images using an        anatomical knowledge dataset; machine learning is used to        improve this dataset.    -   the patient specific anatomic feature is automatically        identified from an analysis of the metadata.    -   analysis of the metadata is done using Natural Language        Processing (NLP).    -   2D medical images are anonymised such that no identifiable        healthcare information is being transferred;    -   a cryptographic hash of the patient information is created;    -   the system uses digital signatures to verify identity and        approve decisions    -   a smart contract object required to order or initiate the        generation of the 3D model contains information about the        requirements of the prints such as: stage quality gates,        insurance status, payment status/limits, clinician, patient        consent, annotations, data sharing agreements and/or data        processing agreements;    -   the smart contract object is incorporated into a Blockchain;    -   the smart contract object is pre-agreed between the patient and        the end-user;    -   printing of the 3D model is only executed once the smart        contract object has been validated;    -   dynamic pricing is generated when the smart contract object is        validated;    -   an instant quotation is displayed;    -   Digital currency is linked to the printing of the 3D model;    -   material used for printing is automatically selected depending        on the specific anatomic feature;    -   texture used for printed is automatically selected depending on        the specific anatomic feature;    -   when the specific anonymised feature cannot be printed by a        single printer, custom 3D models are constructed, printed and        combined into a single [oversized] 3D printed model.    -   end-user can upload data about the model being generated and        annotate the model as it progresses through the workflow;    -   end-user can select the specific anatomy they wish to 3D print.    -   the uploading of the 2D medical images and the ordering process        is intuitive (non technical person can order a 3D model);    -   an audit trail of the printing process is continuously updated        and tracked (authorization etc);    -   anomalies of the 2D medical images are detected, such as:        incorrect classification of medical images, incorrect patient        data, presence of foreign objects in medical images, low quality        imaging data;    -   images which contains unwanted artefacts or background noise,        such as foreign objects or a bed, are still processed;    -   2D medical images (CT/MRI/PET etc) can be processed together;    -   a preview of the specific anatomy 3D model is displayed to the        end-user before the order is completed;    -   real time information on specific timeframe before a print is        ready is generated (taking into consideration the segmentation,        surface conditioning and printing);    -   print scheduling/distributed printing is based on inbound models        and surgical requirement;    -   the 3D printed model is optimized based on the following patient        specific parameters: scan type, bone type, tissue type, age,        gender, weight;    -   patient specific parameters are extracted from the uploaded        data;    -   [Hardlinking] original order and input data is applied to the        physical object—QR code, NFC chip, RFID tag are added to the        printed model;    -   Profile of end-users are saved with their preferences for        particular model types;    -   segmentation of the 2D medical images is performed to classify        each pixel within the medical images;    -   segmentation technique that is used is one of the following:        threshold, decision tree, chained decision forest or neural        network method;    -   segmentation technique that is used is a combination of the        following techniques: threshold, decision tree, chained decision        forest or neural network method.    -   segmentation is done by applying a threshold to generate a set        of 2D threshold images representing a patient specific anatomic        feature;    -   the segmentation step is combined with an anatomic feature        identification algorithm;    -   a 3D surface mesh model is generated for each set of 2D        threshold images;    -   the threshold value is generated from the 2D medical images        histogram analysis;    -   the threshold value is generated from detecting the peaks of        histogram corresponding to tissues similar to the tissue of the        patient specific anatomic feature.    -   the threshold value is a function of the type of 2D medical        images (CT, MRI, PET or SPCET);    -   the threshold value is a function of the CT scanning parameters        (X-Ray energy and flux);    -   the threshold value is a function of the bone type (hard or        soft);    -   the end user selects the scan type or bone type;    -   the threshold value is not selected by an end-user.    -   the 3D surface mesh models are compressed, smoothed and reduced        before sending them back to the end-user device;    -   the generation of the 3D model is performed by parallel        processing;    -   end-user is alerted when an anomaly is detected;    -   one or more threshold values are used in order to generate one        or more 3D surface mesh model;    -   the end-user selects one of the one or more 3D surface mesh        models he wishes to print;

Concept B: UX—Real Time User Interaction

Computer implemented method comprising the steps of: receiving from anend-user a set of 2D medical images specific to a patient, automaticallysegmenting the set of 2D medical images and creating a 3D printablemodel from the set of 2D segmented medical images, and displaying the 3Dmodel to the end-user.

The computer implemented method may further comprise the optionalfollowing steps:

-   a) The user can send a set of 2D medical images specific to a    patient to the Axial3D server through the web application;-   b) A 3D model candidate, generated by the Axial3D image analysis    pipeline identifying the anatomical components within the images, is    used to generate a 3D model of the organs of interest;-   c) the results of the image analysis and the 3D printable models    created are displayed to the user through the Axial3D web    application;-   d) The web application allows the user to provide contextual    information regarding the patient and the prescription;-   e) The web application provides a basic level of functionality to    enable the end-user to interact with, modify the results and select    the 3D automatic image analysis; this includes the selection of    volumes to be printed, select the materials and the anatomical    fidelity level required (the model accuracy), flag for gross    mis-calculations and require a segmentation engineer to review the    results;-   f) The web application allows the user to trigger the generation of    a new model based on the prescription and the annotation provided; a    new model is then displayed;-   g) the clinical staff (customers) can accept the model once    satisfactory results are shown;-   h) Using this semi-automatic approach allows extremely accurate and    high fidelity 3D printable models to be obtained within a few    minutes instead of the several hours it would otherwise require;-   i) the users can select to skip any of the steps described above.

Optional features (each of which can be combined with others) includethe following:

-   -   this semi automatic system relies on the ability to perform        image analysis and anatomical recognition to an exceptional        degree of accuracy within very short timescales (seconds) based        on the user feedback; such feedback can be expressed either as        free text or as actions performed on the 3D models; the task of        performing the image analysis is devolved to the Axial3D medical        image segmentation and recognition (see Concept C).    -   The data is sensitive in nature and covered by privacy law; all        data transmitted across the network is anonymized, encrypted and        compressed prior to transmission; the integrity of all exchanged        data is verified through data checksum;    -   all data exchanged within one single order is covered by a        Blockchain virtual contract; the virtual contract is created        prior to the 2D scan image data is transmitted to the Axial3D        servers, such scan data alongside all segmentation and 3D model        data is also covered by the same virtual contract; acceptance of        the final model will trigger the completion of the virtual        contract;    -   end-user can easily annotate the anatomic feature;    -   end-user can select anatomic feature they wish to 3D print;    -   3D model is 3D printed to represent a 1:1 scale of the specific        anatomic feature;    -   threshold values are generated from the medical images histogram        analysis;    -   threshold values are function of the type of 2D medical images        (CT, MRI, PET or SPCET);    -   threshold values are function of the CT scanning parameters        (X-Ray energy and flux);    -   threshold values are function of the bone type (hard or soft);    -   the end user selects the scan type or bone type;    -   the threshold value is generated from the 2D medical images        histogram analysis;    -   the threshold value is generated from detecting the peaks of        histogram corresponding to tissues similar to the tissue of the        patient specific anatomic feature.    -   the set of 2D medical images is anonymised before being sent to        a server;    -   the server sends back the 3D surface mesh models back to the        end-user device;    -   the 3D surface mesh models are compressed, smoothed and reduced        before sending them back to the end-user device;    -   the generation of the 3D model is performed by parallel        processing;    -   end-user is alerted when an anomaly is detected;    -   end-user can upload data about the model being created and        annotate the model;    -   instant quotation is displayed.

Concept C: Image Processing Method for Converting 2D DICOM Image Seriesinto a 3D Printable Model.

Computer implemented method for automatically converting 2D DICOM imageseries into a 3D printable model in which the method includes anautomatic segmentation step.

Optional features (each of which can be combined with others) includethe following:

-   -   1. Anatomical Tissue Segmentation: obtain accurate and high        fidelity detection of the tissue of interest;        -   a. tissues that are identified are: osseous tissue; fatty            tissue (including liver), epithelial tissue (glands,            kidneys, pancreas); squamous epitelial tissues (skin);            cardiac tissue; tubular tissue (veins, lymphatic vessels);            lungs; cerebral tissue (brain, primarily from MRI);        -   b. The segmentation is performed using multiple segmentation            algorithms (3 thresholds values, Decision Trees, Neural            Nets) and combines their results into one final segmentation            that is both high fidelity and accurate (the NN is covered            by Concept D);    -   2. Anatomical Recognition (Concept E): the tissues are grouped        into organs and a full picture of the body parts is derived (E.G        spleen, liver, part of the lungs and few ribs may be recognised        together within the same scan); the grouping is performed by a        classifying neural network; this NN contains a small number of        convolutional layers and is followed by a fully connected        section;    -   3. Segmentation Correction: once the anatomical organs and        features are correctly identified, minor errors in the results        of the tissue segmentations can be corrected; errors include        primarily separation of tissues in close proximity (joined        bones) and small miss-detected regions;    -   4. Foreign Bodies Identification (Concept E): implants        (orthopedic metal implants, dental implants and fillings, etc),        pacemakers, unknown foreign bodies (swallowed objects);    -   5. Selection of the organs/anatomical features: these are the        features that will eventually be printed; the anatomical        features of interest are derived primarily from the prescription        provided by the clinicians;    -   6. Print Ready Model Generation: the segmented and identified        volumes are converted into a print ready model:        -   a. The 3D surface for the anatomical features of interest is            generated;        -   b. the 3D surface undergoes simple mesh cleaning algorithms            in order to render it printable (decimation, ensure manifold            structures, fill holes within the mesh)        -   c. each volume is assigned a material        -   d. the 3D printing technology is selected        -   e. Dowels and support are added

Both the training of classifiers and the recognition algorithms takeadvantage of the Axial3D database (Concept F) of pre-labelled scans(Ground Truth); the Axial3D database contains the GT itself as well as aGraph Database describing an Ontology of anatomical features; theontology is tailored taking into account 3D medical imaging practiceswithin the radiology and standard 3D medical imaging techniques (seefollowing concept)

Concept D: Specific Neural Network Processing

Computer implemented method for generating a 3D printable model of apatient specific anatomic feature from the patient 2D medical images inwhich the method includes the step of segmenting the patient 2D medicalimages using a neural network trained from a database of existingmedical images.

Optional features (each of which can be combined with others) includethe following:

-   -   the segmentation step is performed automatically;    -   the method is capable of handling several scanning modalities:        in particular CT, MRI and/or PET scans are supported;    -   A variety of tissues are supported: these include osseous        tissue; fatty tissue (including liver), epithelial tissue        (glands, kidneys, pancreas); squamous epitelial tissues (skin);        cardiac tissue; tubular tissue (veins, lymphatic vessels);        lungs; cerebral tissue (brain, primarily from MRI)    -   the neural network includes only convolutional, downsampling and        upsampling layers; the neural network design does not include        any fully connected layer; the technology combines the ideas of        uNET and FCNN in order to obtain the best segmentation in terms        of anatomical fidelity in regards to the edge of the anatomical        components;    -   the results of the neural network segmentation are combined with        the results of other segmentation techniques such as:        threshold-based, decision tree, chained decision forest method;        This is done in order to maximise the anatomical fidelity;    -   The training of the Neural Network is performed by using the        Axial3D Scan

Database containing labelled data (the Ground Truth) and the medicalimaging ontology (Concept E).

Concept E: Anatomical Feature Identification

Computer implemented method for identifying an anatomic feature from amedical image, the method includes:

-   -   (a) classifying each pixel with the medical image by using a        segmentation method;    -   (b) establishing links between the different pixels from the        exploration of a graph database, wherein the graph database        stores information on human anatomy relevant to medical image        scans; and    -   (c) identifying the anatomic feature from the previously        established links.

Optional features (each of which can be combined with others) includethe following:

-   -   segmentation method is automatic    -   method provides a score or probability that the anatomic feature        has been correctly identified.

The method is optionally, further configured to:

-   -   retrieve the reference of all scans in our data-store related to        hips in CT scans;    -   determine the volume of all radius and ulna available in the        scans;    -   determine the histogram of the Hounsfield value of lungs;    -   detect and identify the bones located in close proximity of the        Clavicola;    -   determine the state of the anatomic feature such as healthy, or        pathology;    -   identify the type of anatomic feature such as tissue type or        organ type.    -   extract additional information such as: HU standard deviation or        average value, estimated anatomic feature volume, estimated        anatomic feature surface area;    -   update the graph database.

Concept F: 3D Medical Imaging DB

Computer implemented method for creating a graph database of medicalimages anatomic features, the method comprising the following steps:

a) Store medical images from CT, MRI, PET scans;

b) Store the labels (Ground Truth) for the scan;

c) Populate a graph database based on standard medical ontologiestailored to 3D medical imaging application; the ontology is representedas a series of nodes connected with each other through functions,proximity and anatomical groupings and the frequency of appearing in thesame 2D medical image;

d) Holds links between the ontologies and GT datasets (both the scan andthe label data).

The GT database contains the 2D scan images alongside the labels; eachscan and relative labels data is approved by the biomedical engineersprior to the insertion into the database; each set of scan and relativelabels are derived from:

-   -   Manually labelled data, labelled either by clinical staff and        made available through an open database or by the Axial3D team        of biomedical engineers;    -   Data derived from the automated segmentation pipeline that has        been reviewed and corrected by a biomedical engineer or clinical        staff,    -   Data derived from manually labelled data to which synthetic        transformations have been applied (synthetic Gt);

The Axial3D Medical Imaging DB is used to perform the following tasks:

-   1. retrieve all scans showing a specific anatomical feature;-   2. create generalized models of anatomical features based on the    existing data;-   3. generate the training and testing sets when training image    classifiers algorithms;-   4. support the anatomical entity identifications once a rough    segmentation is performed.

Optional features (each of which can be combined with others) includethe following:

-   -   medical images are 2D medical scans such as CT, MRI and/or PET        scans;    -   a node contains a reference to a medical image with the        corresponding anatomic feature;    -   a node contains a reference to the results of the segmentation        of a medical image with the corresponding anatomic feature;    -   a node also holds informations relating to the anatomic feature        such as volume, surface area, Hounsfield Unit standard deviation        or average.    -   additional metadata associated with the medical image are also        received alongside the medical images;    -   metadata includes: patient's prescriptive information, medical        professional information, patient information.

Concept G: Design of Objects for Bioprinting—Including Creation ofIdealised Version of Anatomical Features

A method for generating 3D models of portions of anatomy based onanatomical database or mirroring for use in replacing pathologicaltissues in the body.

Optional features (each of which can be combined with others) includethe following:

-   (a) identification and classification of anatomy from 2D medical    images; leveraging a knowledge base of healthy tissues and a    connectivity map of the adjacent tissue. This can be used to    classify abnormal anatomy as a deviation from the norm. The use of a    generalised knowledge base of the 3D shapes of anatomy and their    connectivity to identify non-normal features of anatomy.-   (b) the generation of idealized 3D models of patient anatomy based    on a database of known healthy tissues, organs and anatomical    features; the idealised model is patient specific and involves a    number of parameters including age, gender, weight and height as    well as the identification of the analogous mirrored healthy organ    if available;-   (c) a method to register and spatially align the 3D models of pieces    of anatomy. This method can be used to identify differences between    the defective and the idealized models in order to derive the    defective portion;-   (d) a method to derive the defective portion; this method will be    sufficiently robust so that small differences between the 3D models    such as noise and the inherent variability of the human anatomy are    not included in the final defective portion (edges);-   (e) a method for generating the structure of the 3D printable    lattice for final 3D bio-printing; the lattice type and structure    (regular or irregular) and parameters (cell size, density) will be    tissue & patient specific to the portion of anatomy in question that    will be replaced;-   (f) a method to manually review and approve the final defective    portion against idealised anatomy and prepare the model for final 3D    bio-printing.

NOTE

It is to be understood that the above-referenced arrangements are onlyillustrative of the application for the principles of the presentinvention. Numerous modifications and alternative arrangements can bedevised without departing from the spirit and scope of the presentinvention. While the present invention has been shown in the drawingsand fully described above with particularity and detail in connectionwith what is presently deemed to be the most practical and preferredexample(s) of the invention, it will be apparent to those of ordinaryskill in the art that numerous modifications can be made withoutdeparting from the principles and concepts of the invention as set forthherein.

1. A method for generating a 3D physical model of a patient specificanatomic feature from 2D medical images, in which: (a) the 2D medicalimages are uploaded by an end-user via a Web Application and sent to aserver; (b) the server processes the 2D medical images and automaticallygenerates a 3D printable model of a patient specific anatomic featurefrom the 2D medical images using a segmentation technique; and (c) the3D printable model is 3D printed as a 3D physical model such that itrepresents a 1:1 scale of the patient specific anatomic feature.
 2. Themethod of claim 1 in which the 2D medical images are uploaded alongsidemetadata, wherein the metadata is either uploaded or entered by theend-user and includes one or more of the following: patient'sprescription information, medical professional information or any otherpatient-related additional information, and in which metadata can beadded by the end-user in real time via the Web application.
 3. Themethod of claim 1 in which the patient specific anatomic feature isautomatically identified from analysing the 2D medical images usingmachine learning techniques applied to an anatomical knowledge dataset.4. (canceled)
 5. The method of claim 2 in which the patient specificanatomic feature is automatically identified from an analysis of themetadata performed using Natural Language Processing (NLP). 6-8.(canceled)
 9. The method of any claim 1 in which the 3D printable modelis automatically displayed to the end-user via the Web Application, andthe end-user is able review, annotate and/or modify the 3D printablemodel in real time and/or select the specific anatomic feature they wishto 3D print.
 10. (canceled)
 11. The method of claim 1 in which the 2Dmedical images and any additional metadata are anonymised prior to beingsent to the server such that no identifiable healthcare or personalinformation is being transferred to the server.
 12. The method of claim1 in which a cryptographic hash of the patient information is created toenable identification of a specific patient without sharing sensitivepatient information.
 13. The method of claim 1 in which a smart contractobject, required to order or initiate the generation of the 3D physicalmodel, contains information about the requirements of the 3D physicalmodel to be printed including one or more of: stage quality gates,insurance status, payment status/limits, clinician, patient consent,annotations, data sharing agreements and/or data processing agreements;and in which printing of the 3D physical model is only executed once thesmart contract object has been validated.
 14. The method of claim 13 inwhich the smart contract object is incorporated into a Blockchain.15-17. (canceled)
 18. The method of claim 12 in which dynamic pricing isgenerated after the validation of the smart contract object, and aninstant quotation is displayed to the end-user and in which a digitalcurrency is linked to the printing of the 3D physical model. 19.(canceled)
 20. The method of claim 1 in which the material and textureused for printing is automatically selected depending on its resemblanceto the specific anatomic feature.
 21. The method of claim 1 in which thematerial used for printing is automatically selected based on therequirement of the 3D physical model, such as the time required toachieve the printing of the 3D physical model.
 22. (canceled)
 23. Themethod of claim 1 in which the 3D physical model is optimized based onthe following patient related parameters: scan type, bone type, tissuetype, age, gender, weight, in which patient related parameters areextracted from data uploaded or entered via the Web Application. 24.(canceled)
 25. The method of claim 1 in which multiple 3D printablemodels are generated and multiple 3D physical models are printed andcombined when a single printer cannot print the patient anatomic featureand in which one or more connective pieces are printed in order for themultiple 3D physical models to be combined together into a single 3Dphysical model, and the material used for printing the connective piecesis automatically predicted. 26-28. (canceled)
 29. The method of anyclaim 1 in which multiple segmentation techniques are used, and theresults of each segmentation technique are combined together to derive afinal segmentation result in which the segmentation technique is one ora combination of the following techniques: threshold-based, decisiontree, chained decision forest or neural network method.
 30. The methodof claim 1 in which a threshold-based segmentation method is used andthe threshold value is automatically generated from the 2D medicalimages histogram analysis using one or more of the following: thethreshold value is a function of the type of 2D medical images, and/orthe threshold value is a function of the 2D medical images scanningparameters such as X-Ray energy and/or flux, and/or the threshold valueis optimised based on one or more of the following parameters: scantype, bone type, tissue type, age, gender and weight of the patient,and/or the threshold value is generated from detecting the peaks of the2D medical images histogram corresponding to tissues similar to thetissue of the patient specific anatomic feature.
 31. (canceled)
 32. Themethod of claim 1 in which the segmentation technique uses a logistic orprobabilistic function to calculate the likelihood of a pixel of beingthe tissue corresponding to the patient specific anatomic feature. 33.The method of claim 1 in which a threshold-based segmentation method isused in combination with a pre-processing filter such as a Gaussianfilter. 34-38. (canceled)
 39. The method of claim 1, in which athreshold-based segmentation technique is used and the threshold valueis generated from detecting the peaks of the 2D medical images histogramcorresponding to tissues similar to the tissue of the patient specificanatomic feature, the threshold-based segmentation technique furthercomprises the following steps performed for each detected peak of the 2Dmedical images histogram: (a) deriving or determining a detected peakinflection point by calculating the zero of the medical images histogramsecond derivative in proximity of a peak; (b) deriving or determining aninflection offset between the peak and the inflection point; (c)estimating a threshold corresponding to the position of the detectedpeak with an offset corresponding to three time the derived ordetermined inflection offset.
 40. (canceled)
 41. The method of claim 1in which a threshold-based segmentation is used and multiple thresholdsare applied to the 2D medical images such that multiple 3D printablemodels are automatically generated.
 42. The method of claim 1 in whichthe segmentation technique uses a decision tree trained using existingpre-labelled medical images, in which the following properties of the 2Dmedical images pixels are taken into account in order to create thedecision tree: number of pixels resembling the tissue of interestlocated near the pixel in question, number of pixels resembling thetissue of interest located near a given pixel, or overall gradient ofthe image at a given pixel.
 43. (canceled)
 44. The method of claim 42 inwhich the decision tree is applied to a subset of pixel within the 2Dmedical images and the labels obtained from this subset are then upscaled using standard interpolation methods in order to recover thesegmentation of the full image and in which the subset of pixel isgenerated by subsampling the 2D medical images, and in which asubsampling stride is selected depending on the pixel size. 45.(canceled)
 46. The method of claim 1 in which the segmentation techniqueuses a chained decision forest, in which a hierarchy of decision forestsis used, and in which the results of a decision tree and the resultsfrom another segmentation technique are fed to a new decision treealongside the pixel values of the 2D medical images, in which eachforest-node is treated as a simple classifier that produces a score asto how likely a pixel is to belong to the tissue corresponding to thespecific patient anatomic feature. 47-48. (canceled)
 49. The method ofclaim 1 in which the segmentation technique uses a Neural Networkmethod, such as a Fully Convolutional Neural Network (FCNN) or a UNETNeural network, in which the Neural Network is trained from a databaseof existing medical images that have been labelled and a medical imagingontology. 50-51. (canceled)
 52. The method of claim 49 in which theNeural Network includes only convolutional, downsampling and upsamplinglayers.
 53. The method of claim 49 in which the Neural Network does notinclude any fully connected layer and combines the ideas of uNET andFCNN in order to obtain an optimised segmentation in terms of anatomicalfidelity regarding the edge of the anatomic feature.
 54. The method ofclaim 49 in which upsampling layers are added and in which the outputsof previous layers are used to identify regions of the 2D medical imagesin order to lead to a specific classification.
 55. (canceled)
 56. Themethod of claim 1 in which the segmentation of the 2D medical images isperformed to classify each pixel within the 2D medical images and iscombined with an anatomic feature identification algorithm that uses agraph database of medical images anatomic features, in which the methodfurther includes establishing links between the different classifiedpixels from the exploration of the graph database, and identifying thepatient specific anatomic feature from the established links. 57-59.(canceled)
 60. The method of claim 56 in which the graph databasecomprises nodes representing anatomic features such as tissue typeand/or organ type, and edges associated with the relationships betweenthe nodes, such as: has part, proximity, attachment, ligament,functional, and in which a node includes: a reference to a medical imagewith the corresponding anatomic feature, a reference to the results ofthe segmentation of a medical image with the corresponding anatomicfeature, information relating to the anatomic feature such as volume,surface area, Hounsfield Unit standard deviation or average, and inwhich the graph database is updated after the generation of a 3Dprintable model. 61-62. (canceled)
 63. The method of claim 1 in which ascore or probability that the anatomic feature has been correctlyidentified is provided.
 64. The method of claim 1, in which the methodfurther includes a feature extraction algorithm that takes advantage ofboth the output of the segmentation technique as well as the as the 2Dmedical images data in order to obtain interesting properties of thepatient specific anatomic feature, such as one or more of the following:volume, surface area, Hounsfield unit, standard deviation acrossavailable medical images, histogram of the Hounsfield Unitscorresponding to the anatomic feature across an anatomical knowledgedataset or the smallest bounding box containing the patient specificanatomic feature.
 65. (canceled)
 66. The method of claim 64 in which thefeature extraction algorithm is further used to extract one or more ofthe following interesting properties: presence of specific keypointlandmarks, a number of predefined shapes and volumes within the anatomicfeature being considered or any specific properties that are unique tothe specific anatomical feature, and in which the extracted interestingfeatures are then added to the graph database as part of the nodeproperties.
 67. (canceled)
 68. The method of claim 64, in which theextracted interesting features can be used in order to derive aclassification of the anatomical components located within the 2Dmedical images, using the following steps: (a) derive segmentation usingone or more automated segmentation techniques; (b) apply the featureextraction algorithm(s) to the segmentation in order to derive thevalues of the extracted interesting features; (c) compare to an existingdataset of interesting features and find a number of potential matches;(d) filter the matches depending on a proximity map derived from thegraph database, refine the filtering using a standard model andcalculate a score or probability of having matched the interestingfeature correctly; (e) use the calculated score or probability obtainedin a decision tree or forest tree in order to derive a finalclassification for a specific tissue or organ; and in which anydeviation from the standard model is detected.
 69. (canceled)
 70. Themethod of claim 1 in which touching organs or tissues are detectedwithin the 2D medical images and an edge finding algorithm is used toseparate the different tissues or organs, and.
 71. (canceled)
 72. Themethod of claim 1 in which deformities and/or pathology of the anatomicfeature are detected by measuring the deviation from a normal or healthyappearance of the anatomic feature, in which the method is used forgenerating a 3D bio-compatible physical model of a patient specificanatomic feature or a portion of a patient specific anatomic feature, inwhich the automatically segmented data is assessed against statisticalmodel of pre-segmented anatomy ‘Best fit model’, and a 3D printablemodel is created based on a statistical model for patients' anatomy toinsure an optimal reconstruction of tissue, and in which missingfragments are predetermined with a best fit model and tissue scaffoldmodels created from this.
 73. (canceled)
 74. The method of claim 1 inwhich a 3D surface mesh model of the patient specific anatomic featureis generated from the segmented 2D medical images, in which the 3Dsurface mesh is extracted from the scalar volume data, processed by amesh cleaning algorithm, compressed, smoothed and reduced before beingsent back to the end-user via the Web-Application. 75-77. (canceled) 78.The method of claim 74 in which the 3D surface mesh model is 3Dprintable and has the following properties: all disjointed surfaces areclosed manifolds; appropriate supports are used to keep the disjointedsurfaces/volumes in place, appropriate supports are used in order tofacilitate 3D printing; if a hollow volume is specifically requested byan end-user: appropriate drainage holes are added manually by anoperation team.
 79. The method of claim 74 in which a 3D model of thesurface of the patient specific anatomic feature is extracted from the3D mesh model, and in which a marching cube algorithm is used in orderto force some vertices to be placed on voxels that do not intersect aniso-surface directly.
 80. (canceled)
 81. The method of claim 74, inwhich the method further includes a surface conditioning step, thesurface conditioning step including the steps of: (a) applyingpoly-reduction algorithms to the 3D mesh model, (b) correcting errorssuch as duplicated points, overlapping surfaces or missing surface toensure the mesh is a manifold; (c) applying a mesh filter; (d) detectingand covering holes; (e) selecting appropriate textures.
 82. (canceled)83. The method of claim 74, in which the method further includes thefollowing steps of: (a) filling watertight surfaces; (b) adding dowelsto support the printing of a specific anatomic feature; (c) determiningall local minima of the surface and adding print supports; (d) removingthe print supports during printing post processing.
 84. (canceled) 85.The method of claim 1 in which one or more 3D printable models is sentto the end-user via the Web-application.
 86. (canceled)
 87. The methodof claim 1 in which the method is configured to detect an anomaly withinthe 2D medical images, such as: incorrect classification of medicalimages, incorrect patient data, presence of foreign objects or lowquality imaging data, and in which the end-user is alerted when ananomaly is detected.
 88. (canceled)
 89. The method of claim 1 in whichthe method is able to handle 2D medical images which include unwantedartefacts or background noise, such as foreign objects or a bed.
 90. Themethod of claim 1 in which a preview of the 3D printable model isdisplayed to the end-user for approval before printing the 3D physicalmodel.
 91. The method of claim 1 in which information on an expectedtimeframe to generate a 3D physical model is calculated and displayed tothe end-user in real time and in which the expected timeframe takes intoconsideration one or more of the following: segmentation, surfaceconditioning and printing phases. 92-94. (canceled)
 95. The method ofclaim 1 in which the 2D medical images and any additional metadata arehard linked to the 3D physical model via a QR code, NFC chip or RFIDtag. 96-99. (canceled)
 100. A 3D physical model representing a 1:1 scaleof a patient specific anatomic feature that is generated from the methodof claim
 1. 101. A Computer implemented system for generating a 3Dprinted model of a patient specific anatomic feature from 2D medicalimages, the system comprising: (a) an interface module configured toreceive 2D medical images and to send the 2D medical images to a server,(b) a server configured to process the 2D medical images andautomatically generate a 3D printable model of a patient specificanatomic feature from the 2D medical images using a segmentationtechnique; and (c) a 3D printer configured to receive the 3D printablemodel and to 3D print a 3D physical model such that it represents a 1:1scale of the patient specific anatomic feature.